fix(eval): surface batch-evaluation result truncation as a warning - #1964
Merged
Conversation
…ntly returning partial results When readEvaluationResults hits MAX_RESULT_PAGES it now throws instead of warning to the file logger and returning the accumulated list. getBatchEvaluation already catches that into resultsError, which the CLI surfaces as a stderr warning (stdout metadata stays clean) — the same customer-visible path as any other CloudWatch read failure. Previously 'get --json' succeeded with partial 'results' and no customer-visible signal that they were incomplete. Follow-up to #1924 (aidandaly24 review comment).
github-actions
Bot
added
the
agentcore-harness-reviewing
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## refactor #1964 +/- ##
============================================
+ Coverage 96.71% 96.74% +0.02%
============================================
Files 296 296
Lines 16209 16213 +4
============================================
+ Hits 15677 15685 +8
+ Misses 532 528 -4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
github-actions
Bot
removed
the
agentcore-harness-reviewing
Contributor
Author
Live demo against the exploratory accountVerified the new warning path end-to-end. To force truncation on a real job I temporarily set 1. Truncation now warns (stderr) while the job metadata still prints (stdout): $ bun run src/index.ts eval batch-evaluation get --id BugBash3P_BugBash3P_1785443436425-b1c354f3b9 --region us-east-1 | head -12
warning: could not retrieve CloudWatch results (batch-evaluation results exceed 1 CloudWatch pages; retrieved 5 results are incomplete). Job status is unaffected. See CloudWatch: region us-east-1, log group /aws/bedrock-agentcore/evaluations/batch-evaluations/results/default, stream run-BugBash3P_BugBash3P_1785443436425-b1c354f3b9.
{
"batchEvaluationId": "BugBash3P_BugBash3P_1785443436425-b1c354f3b9",
"batchEvaluationArn": "arn:aws:bedrock-agentcore:us-east-1:725476964917:batch-evaluate/BugBash3P_BugBash3P_1785443436425-b1c354f3b9",
"batchEvaluationName": "BugBash3P_BugBash3P_1785443436425",
"status": "COMPLETED",
...2. The warning goes to stderr only — stdout stays clean, machine-readable JSON, and $ bun run src/index.ts eval batch-evaluation get --id ... --region us-east-1 2>/dev/null | python3 -c 'import sys,json; d=json.load(sys.stdin); print("stdout is valid JSON. status:", d["status"], "| results key present:", "results" in d)'
stdout is valid JSON. status: COMPLETED | results key present: FalseThis is the intended behavior: truncation travels the same customer-visible |
added 2 commits
August 10, 2026 18:57
…e cap Replace the bare Error with a new AgentCoreCLIError subclass, ResultTruncationError (source: INTERNAL — the page cap is ours, not a user or service fault). Carries the log group/stream, cap, and retrieved count in meta for telemetry. Slots into the existing errors hierarchy alongside NetworkingError/FileWriteError; getBatchEvaluation's try/catch and the stderr warning path are unchanged.
The message already carries the page count and retrieved count; the log group/stream are in the handler's stderr warning. No separate meta payload.
aidandaly24
approved these changes
Aug 10, 2026
nborges-aws
approved these changes
Aug 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Follow-up to #1924. Addresses @aidandaly24's non-blocking review comment on
src/core/batchEvaluationResults.tsx:Change
readEvaluationResultsnow throws when it exhaustsMAX_RESULT_PAGESwith the forward token still advancing (i.e. the stream has more pages than we read), instead of logging to the file logger and returning the partial list.getBatchEvaluationalready wraps the call in atry/catchthat routes any throw intoresultsError, and thegethandler already turnsresultsErrorinto a stderr warning viawarnCloudWatchFailurewhile leaving stdout metadata clean. So truncation now travels the same customer-visible path as every other CloudWatch read failure — no new plumbing.Before:
get --jsonon a very large job returned partialresultsthat read as complete, with only a file-logger line.After: the job metadata still prints on stdout; a stderr warning states the results are incomplete.
Test
Added a unit test with an ever-advancing token so the loop runs into the cap; asserts
readEvaluationResultsrejects rather than returning a partial list. Existing pagination/parse tests unchanged and green (bun test src/core/batchEvaluationResults.test.ts→ 6 pass; handler tests → 12 pass).